In order to reduce the matching error rate of ORB (Oriented FAST and Rotated BRIEF) method caused by the scale invariance of the feature points in the algorithm and enhance the robustness of the descriptors of Binary Robust Independent Elementary Features (BRIEF) algorithm to noise, an improved feature point matching method was proposed. Speeded-Up Robust Features (SURF) algorithm was used to extract feature points, and BRIEF algorithm with direction information was used to describe the feature points. Random pixel pairs in the neighborhood of the feature point were selected, the comparison results of the grayscales and the similarity of pixel pairs were encoded respectively, and Hamming distance was used to calculate the differences between the two codes. The similarity between the feature points were measured by the adaptive weighted fusion method. Experimental results show that the improved method has better adaptability to the scale variance, illumination variance and blurred variance of images, can obtain a higher feature point correct matching rate compared with the conventional ORB method, and can be used to improve the performance of image stitching.